Raise an error when the specified OpenSSL library directory doesn't exist. #618

junaruga · 2023-05-03T19:36:05Z

This pull-request fixes #617. At least it gives users an instruction in the case of rake compile -- --with-openssl-dir=/path/to/openssl below.

$ ls -l $HOME/.local/openssl-3.0.8-fips-debug-trace
total 20
drwxr-xr-x. 2 jaruga jaruga 4096 Apr 25 16:43 bin/
drwxr-xr-x. 3 jaruga jaruga 4096 Apr 25 16:43 include/
drwxr-xr-x. 5 jaruga jaruga 4096 Apr 25 16:43 lib64/
drwxr-xr-x. 4 jaruga jaruga 4096 Apr 25 16:45 share/
drwxr-xr-x. 5 jaruga jaruga 4096 May  2 18:24 ssl/

$ bundle exec rake compile -- --enable-debug --with-openssl-dir=$HOME/.local/openssl-3.0.8-fips-debug-trace
mkdir -p tmp/x86_64-linux/openssl/3.2.1
cd tmp/x86_64-linux/openssl/3.2.1
/usr/local/ruby-3.2.1/bin/ruby -I. -r.rake-compiler-siteconf.rb ../../../../ext/openssl/extconf.rb -- --enable-debug --with-openssl-dir=/home/jaruga/.local/openssl-3.0.8-fips-debug-trace
*** ../../../../ext/openssl/extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=../../../../ext/openssl
	--curdir
	--ruby=/usr/local/ruby-3.2.1/bin/$(RUBY_BASE_NAME)
	--with-openssl-dir
	--with-openssl-include
	--without-openssl-include=${openssl-dir}/include
	--with-openssl-lib
	--without-openssl-lib=${openssl-dir}/lib
../../../../ext/openssl/extconf.rb:21:in `<main>': OpenSSL library directory could not be found in '/home/jaruga/.local/openssl-3.0.8-fips-debug-trace/lib'. You might want to use --with-openssl-lib=<dir> option to specify the directory. (RuntimeError)
rake aborted!
Command failed with status (1): [/usr/local/ruby-3.2.1/bin/ruby -I. -r.rake...]
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.2.0/gems/rake-compiler-1.2.1/lib/rake/extensiontask.rb:220:in `block (2 levels) in define_compile_tasks'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.2.0/gems/rake-compiler-1.2.1/lib/rake/extensiontask.rb:217:in `block in define_compile_tasks'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/ruby-3.2.1/bin/bundle:25:in `load'
/usr/local/ruby-3.2.1/bin/bundle:25:in `<main>'
Tasks: TOP => compile => compile:x86_64-linux => compile:openssl:x86_64-linux => copy:openssl:x86_64-linux:3.2.1 => tmp/x86_64-linux/openssl/3.2.1/openssl.so => tmp/x86_64-linux/openssl/3.2.1/Makefile
(See full trace by running task with --trace)

Commit message

Below is the commit message.

Add a logic to check a OpenSSL library directory existence on the given --with-openssl-dir=<dir> option.

OpenSSL creates the library directory to the /path/to/openssl_dir/lib64 as a default, when it is built from the source as follow.

OpenSSL

$ ./Configure \
  --prefix=/path/to/openssl
$ make
$ make install

In the case of the following command, the dir_config("openssl") returns ["/path/to/openssl/include", "/path/to/openssl/lib"]. And this causes the Ruby OpenSSL binding is unintentionally built with the system OpenSSL's library directory, because the /path/to/openssl/lib doesn't exist. The logic to check the OpenSSL library directory's existence is to avoid building in this case.

ruby/openssl

$ bundle exec rake compile -- \
  --with-openssl-dir=/path/to/openssl

In the case of the following command, the dir_config("openssl") returns ["/path/to/openssl/include", "/path/to/openssl/lib64:/path/to/openssl/lib64:/path/to/openssl/lib"]. The returned library directory string is a set of the directories with the path separator ":".

$ bundle exec rake compile -- \
  --with-openssl-dir=/path/to/openssl \
  --with-openssl-lib=/path/to/openssl/lib64

ext/openssl/extconf.rb

junaruga · 2023-07-17T21:29:40Z

Rebasing on the latest master branch. I also updated the commit message too.

junaruga · 2023-07-17T21:43:42Z

I see one warning in macOS truffleruby-head case.

https://github.com/ruby/openssl/actions/runs/5580792815/jobs/10198185535#step:8:94

../../../../ext/openssl/openssl_missing.h:195:11: error: 'TS_VERIFY_CTS_set_certs' macro redefined [-Werror,-Wmacro-redefined]
 #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
          ^
/usr/local/Cellar/openssl@3/3.1.1_1/include/openssl/ts.h:426:11: note: previous definition is here
#  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
          ^

junaruga · 2023-07-17T23:04:50Z

@rhenium what do you think? Would you like reviewing? I plan to merge it if it looks good to you.

rhenium · 2023-07-18T07:07:17Z

This is a common pitfall and a check can be certainly useful: if the lib directory derived from --with-openssl-dir doesn't exist, there is definitely an issue.

However, I'm unsure whether suggesting the use of --with-openssl-lib= is appropriate. I think the decision of whether to use lib or lib64 (or any other name?) should be made by the system admin or the distro, and it should be consistent across all software installations in the same system.

While using --with-openssl-lib= would work in this case, I suspect the "correct" solution is either:

recompiling OpenSSL with --libdir=#{File.basename(RbConfig::MAKEFILE_CONFIG["libdir"])}, or
recompiling Ruby with --with-libdir=<whatever OpenSSL uses>.

junaruga · 2023-07-19T10:59:38Z

Thanks for your comment.

However, I'm unsure whether suggesting the use of --with-openssl-lib= is appropriate. I think the decision of whether to use lib or lib64 (or any other name?) should be made by the system admin or the distro, and it should be consistent across all software installations in the same system.

I agree with the decision of using lib or lib64 (or any other name) should be made by the system admin, or distros. And I agree it should be consistent in the system. That means that users don't need to use the --with-openssl-dir= argument to compile Ruby OpenSSL binding with specific OpenSSL library in a normal case.

And it seems that the admin or distros can configure the search directories that is used as gcc -L <dir> in the system. (reference) So, if admin configure the gcc (or clang or any C compilers) search path properly, users don't need to set --with-openssl-dir= to compile Ruby OpenSSL binding.

While using --with-openssl-lib= would work in this case, I suspect the "correct" solution is either:

recompiling OpenSSL with --libdir=#{File.basename(RbConfig::MAKEFILE_CONFIG["libdir"])},

I don't understand this part. You meant that recompiling "Ruby" OpenSSL binding (ruby/openssl) with Ruby library directory such as /path/to/ruby_dir/lib/? Or recompiling OpenSSL (openssl/openssl) with Ruby library directory?

or
recompiling Ruby with --with-libdir=.

I agree with this, as ruby/openssl can be used as a standard library of the Ruby.

Considering your comment, how can I change this PR or the commit message or I don't need to change the PR?

rhenium · 2023-07-21T18:48:00Z

recompiling OpenSSL with --libdir=#{File.basename(RbConfig::MAKEFILE_CONFIG["libdir"])},

I don't understand this part. You meant that recompiling "Ruby" OpenSSL binding (ruby/openssl) with Ruby library directory such as /path/to/ruby_dir/lib/? Or recompiling OpenSSL (openssl/openssl) with Ruby library directory?

or
recompiling Ruby with --with-libdir=.

I agree with this, as ruby/openssl can be used as a standard library of the Ruby.

I meant OpenSSL (openssl/openssl) and Ruby (ruby/ruby) need to agree on the library directory name. File.basename(RbConfig::MAKEFILE_CONFIG["libdir"]) would evaluate to lib in the commit message's system. <whatever OpenSSL uses> in that system is lib64.

From openssl's (ruby/openssl) point of view, it doesn't know which is using the "wrong" library directory name, Ruby or OpenSSL.

On second thought, --with-openssl-lib does actually have use-cases and I think it's useful to mention it.

--with-openssl-lib should be used, for example, when building against Arch Linux's openssl-1.1 package. This package installs header files in /usr/include/openssl-1.1 and /usr/lib/openssl-1.1. The only option here is to explicitly specify --with-openssl-include and --with-openssl-lib.

#477 (comment) is another example with MacPorts, which should have been obvious with the warning added by this PR.

However, the commit message doesn't match. If custom-built OpenSSL is using the different library directory name from the rest of the system, the correct solution is to recompile OpenSSL with ./config --libdir=lib.

I started to think we should expand CONTRIBUTING.md or the GitHub wiki pages. CONTRIBUTING.md uses --libdir=lib without an explanation.

junaruga · 2023-07-24T15:28:50Z

I meant OpenSSL (openssl/openssl) and Ruby (ruby/ruby) need to agree on the library directory name. File.basename(RbConfig::MAKEFILE_CONFIG["libdir"]) would evaluate to lib in the commit message's system. in that system is lib64.

From openssl's (ruby/openssl) point of view, it doesn't know which is using the "wrong" library directory name, Ruby or OpenSSL.

Thanks for explaining it.
Right. Now I understand this is something like suggesting it with the following message in the Rakefile.

ruby_ldir_name = File.basename(RbConfig::MAKEFILE_CONFIG["libdir"])
msg += "\n  1. Recompile OpenSSL by configuring with --libdir=#{ruby_ldir_name} option."

recompiling Ruby with --with-libdir=.

I still don't understand this. The ruby/ruby doesn't have the --with-libdir configure option. Could you provide a command lines example to explain this?

$ ./autogen.sh
$ ./configure --with-libdir=foo
configure: error: cannot find required auxiliary files: config.guess config.sub

On second thought, --with-openssl-lib does actually have use-cases and I think it's useful to mention it.

--with-openssl-lib should be used, for example, when building against Arch Linux's openssl-1.1 package. This package installs header files in /usr/include/openssl-1.1 and /usr/lib/openssl-1.1. The only option here is to explicitly specify --with-openssl-include and --with-openssl-lib.

#477 (comment) is another example with MacPorts, which should have been obvious with the warning added by this PR.

However, the commit message doesn't match. If custom-built OpenSSL is using the different library directory name from the rest of the system, the correct solution is to recompile OpenSSL with ./config --libdir=lib.

Okay. I understand that both your Arch Linux and also MacPorts cases explain showing both --with-openssl-include and --with-openssl-lib. in the error message is better than showing --with-openssl-dir and --with-openssl-lib in the message to guide the users.

By the way, you showed the ./config script to configure OpenSSL. I was using ./Configure script instead of that. Seeing the document, https://github.com/openssl/openssl/blob/master/INSTALL.md#building-openssl, I can see ./Configure and perl Configure to configure. I think the error message can be Recompile OpenSSL by configuring with --libdir=#{ruby_ldir_name} option. not showing the exact configure script file name.

I started to think we should expand CONTRIBUTING.md or the GitHub wiki pages. CONTRIBUTING.md uses --libdir=lib without an explanation.

That sounds good!

junaruga · 2023-07-24T15:44:48Z

I started to think we should expand CONTRIBUTING.md or the GitHub wiki pages. CONTRIBUTING.md uses --libdir=lib without an explanation.

I start to think perhaps the following way that is to explain the actual ways in a document page. Because we don't want to modify the extconf.rb as much as possible right? And I think this error can be raised not only contributing but also just installing openssl gem like gem install openssl -- --with-openssl-dir=/opt/openssl.

What do you think?

msg = "OpenSSL library directory could not be found in '#{ssl_ldir}'. " \
  'See <a link of the document> to fix this error.'
raise msg

junaruga · 2023-07-24T15:58:49Z

I rebased this PR on the latest master branch, updating the commit message too. I intent I replace the <link> with the correct link after we create the document.

$ bundle exec rake compile -- \
  --with-openssl-dir=/home/jaruga/.local/openssl-master-lib64-3d2f96e2c
...
../../../../ext/openssl/extconf.rb:30:in `<main>': OpenSSL library directory could not be found in '/home/jaruga/.local/openssl-master-lib64-3d2f96e2c8/lib'. See <link> to fix this error. (RuntimeError)
rake aborted!
...

junaruga · 2023-07-25T14:13:34Z

I still don't understand this. The ruby/ruby doesn't have the --with-libdir configure option. Could you provide a command lines example to explain this?

All right. Now I understood the Ruby configuring case above.

$ ./autogen.sh
$ ./configure --prefix="$(pwd)/dest" --libdir="$(pwd)/dest/lib64"
$ make
$ make install

Then the --with-openssl-dir=/path/to/openssl_dir should check the lib64 case according to the source mkmf.rb#dir_config calling the _libdir_basename method.

$ ./dest/bin/irb 
irb(main):001:0> File.basename(RbConfig::MAKEFILE_CONFIG["libdir"])
=> "lib64"
irb(main):002:0> RbConfig::MAKEFILE_CONFIG["libdir"]
=> "$(exec_prefix)/lib64"

…xist. OpenSSL built from the source creates the library directory to the `/path/to/openssl_dir/lib64` as a default. In the case, the `bundle exec rake compile -- --with-openssl-dir=<openssl_dir>` cannot compile with the lib64 directory, and may compile with system OpenSSL's libraries unintentionally. This commit is to check this case to avoid linking with an unintentional library directory.

junaruga · 2023-07-25T14:52:56Z

I rebased again without mentioning See <a link of the document> to fix this error. as we don't have the link of the document right now.

Now the error message is like this. It seems the (RuntimeError) automatically appended to the 1st line's end.

$ bundle exec rake compile -- --with-openssl-dir=$HOME/.local/openssl-master-lib64-3d2f96e2c8
...
../../../../ext/openssl/extconf.rb:33:in `<main>': OpenSSL library directory could not be found in '/home/jaruga/.local/openssl-master-lib64-3d2f96e2c8/lib'. You might want to fix this error in one of the following ways. (RuntimeError)
  * Recompile OpenSSL by configuring it with --libdir=lib  to specify the OpenSSL library directory.
  * Recompile Ruby by configuring it with --libdir=<dir> to specify the Ruby library directory.
  * Compile this openssl gem with --with-openssl-include=<dir> and --with-openssl-lib=<dir> options to specify the OpenSSL include and library directories.
rake aborted!
...

I referred to the following code for the style of the --somethingdir=<dir>.

openssl/ext/openssl/extconf.rb

Lines 109 to 111 in ee03210

 raise "OpenSSL library could not be found. You might want to use " \ 

 "--with-openssl-dir=<dir> option to specify the prefix where OpenSSL " \ 

 "is installed."

rhenium · 2023-08-04T10:22:35Z

recompiling Ruby with --with-libdir=.

I still don't understand this. The ruby/ruby doesn't have the --with-libdir configure option. Could you provide a command lines example to explain this?

You're right, I must have been confused with another project. --with-libdir= is not a valid parameter for Ruby's configure, but --libdir= is.

rhenium · 2023-08-04T10:44:45Z

By the way, you showed the ./config script to configure OpenSSL. I was using ./Configure script instead of that. Seeing the document, https://github.com/openssl/openssl/blob/master/INSTALL.md#building-openssl, I can see ./Configure and perl Configure to configure. I think the error message can be Recompile OpenSSL by configuring with --libdir=#{ruby_ldir_name} option. not showing the exact configure script file name.

config script was a convenient wrapper around Configure script. Configure used to require the user to specify the target platform, e.g., ./Configure <other options> linux-x86_64. With config, the linux-x86_64 part was automatically guessed.

Looking at the history of INSTALL.md (openssl/openssl@16b0e0f), it seems that Configure in OpenSSL >= 3.0 does this by default and they are completely equivalent, and Configure is now the recommended way. I just learned this, thanks for the information.

junaruga · 2023-08-04T15:51:55Z

All right! Thanks for explaining it.

Do you have any other comments about this PR? Perhaps can we merge it?

rhenium · 2023-08-08T15:11:50Z

Sorry for the slow turn around. It looks good to me. Let's merge it!

junaruga · 2023-08-08T16:01:40Z

It's all right. Thank you for checking and merging this PR!

junaruga commented May 3, 2023

View reviewed changes

ext/openssl/extconf.rb Show resolved Hide resolved

junaruga force-pushed the wip/check-ssl-lib-dir branch from 94da3c0 to 8f7e326 Compare May 3, 2023 19:49

junaruga force-pushed the wip/check-ssl-lib-dir branch from 8f7e326 to fc2ddf5 Compare July 17, 2023 21:29

junaruga changed the title ~~Check a OpenSSL library directory existence.~~ Raise an error when the specified OpenSSL library directory doesn't exist. Jul 17, 2023

junaruga mentioned this pull request Jul 17, 2023

warning: macOS truffleruby-head: TS_VERIFY_CTS_set_certs macro redefined in OpenSSL 3.1 #650

Closed

junaruga requested a review from rhenium July 17, 2023 23:03

junaruga force-pushed the wip/check-ssl-lib-dir branch from fc2ddf5 to c428080 Compare July 24, 2023 15:57

junaruga force-pushed the wip/check-ssl-lib-dir branch from c428080 to ca54087 Compare July 25, 2023 14:48

junaruga mentioned this pull request Jul 26, 2023

GCC -L<openssl lib directory> is wrong with the OpenSSL installed from the souce without --libdir=lib option #617

Closed

rhenium approved these changes Aug 8, 2023

View reviewed changes

rhenium merged commit 9391b56 into ruby:master Aug 8, 2023
40 checks passed

junaruga deleted the wip/check-ssl-lib-dir branch August 8, 2023 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raise an error when the specified OpenSSL library directory doesn't exist. #618

Raise an error when the specified OpenSSL library directory doesn't exist. #618

junaruga commented May 3, 2023

junaruga commented Jul 17, 2023

junaruga commented Jul 17, 2023 •

edited

Loading

junaruga commented Jul 17, 2023 •

edited

Loading

rhenium commented Jul 18, 2023

junaruga commented Jul 19, 2023 •

edited

Loading

rhenium commented Jul 21, 2023

junaruga commented Jul 24, 2023

junaruga commented Jul 24, 2023

junaruga commented Jul 24, 2023 •

edited

Loading

junaruga commented Jul 25, 2023

junaruga commented Jul 25, 2023 •

edited

Loading

rhenium commented Aug 4, 2023

rhenium commented Aug 4, 2023

junaruga commented Aug 4, 2023

rhenium commented Aug 8, 2023

junaruga commented Aug 8, 2023

Raise an error when the specified OpenSSL library directory doesn't exist. #618

Raise an error when the specified OpenSSL library directory doesn't exist. #618

Conversation

junaruga commented May 3, 2023

Commit message

junaruga commented Jul 17, 2023

junaruga commented Jul 17, 2023 • edited Loading

junaruga commented Jul 17, 2023 • edited Loading

rhenium commented Jul 18, 2023

junaruga commented Jul 19, 2023 • edited Loading

rhenium commented Jul 21, 2023

junaruga commented Jul 24, 2023

junaruga commented Jul 24, 2023

junaruga commented Jul 24, 2023 • edited Loading

junaruga commented Jul 25, 2023

junaruga commented Jul 25, 2023 • edited Loading

rhenium commented Aug 4, 2023

rhenium commented Aug 4, 2023

junaruga commented Aug 4, 2023

rhenium commented Aug 8, 2023

junaruga commented Aug 8, 2023

junaruga commented Jul 17, 2023 •

edited

Loading

junaruga commented Jul 17, 2023 •

edited

Loading

junaruga commented Jul 19, 2023 •

edited

Loading

junaruga commented Jul 24, 2023 •

edited

Loading

junaruga commented Jul 25, 2023 •

edited

Loading